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DETAILED ACTION 



1 . This application has been examined. The amendment filed 9/17/04 has been 
entered. 

2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-38 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Strubbe et al (6731307) and Geddes (6751661 ). 

4. Regarding claim 1 , Strubbe et al show the multi modal conversational computer 
system (Figures 1,3, column 7 lines 12-20), with a user interface subsystem configured 
to input multi modal data from an environment (Figures 1 , 3, column 7 lines 20-40), the 
data including data associated with first and second modality input sensor (column 10 
lines 35-55), and the environment including devices controllable by the system (column 
10 lines 5-33); a processor coupled to the system and configured to receive a portion of 
the input data and make a determination of a focus (column 10 lines 57-67) and mood 
(column 12 lines 10-25) of a user, and executes actions in the environment accordingly 
(column 9 lines 40-60). A memory stores the results for subsequent determination 
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(column 10 lines 55-60, column 17 lines 55-67). Strubbe et al do not go into the details 
of determining the user intent per se, but do determine various user cues and mood to 
determine an object or topic to which a user is referring (column 17 lines 1-14). 
Furthermore, Geddes determines intent with user cues to determine an object or topic 
to which a user is referring (column 7 lines 35-65). It would have been obvious to a 
person with ordinary skill in the art to determine the user's intent in the system of 
Strubbe et al, because it would provide a convenient way using user cues to determine 
an object or topic to which a user is referring. 

5. Regarding claim 2, the intent determination in Geddes includes resolving 
referential ambiguity associated with one or more users in the environment based on a 
portion otthe received data (Geddes column 8 lines 17-30). The obviousness to 
combine into Strubbe et al follows as in paragraph 4 of this Office Action. 

6. Regarding claim 3, the intent determination in Geddes includes resolving 
referential ambiguity associated with one or more devices in the environment based on 
a portion of the received data (Geddes column 7 lines 20-27). The obviousness to 
combine into Strubbe et al follows as in paragraph 4 of this Office Action. 

7. Regarding claim 4, in addition to that mentioned for claim 1 , Strubbe et al also 
show the execution of the actions includes controlling devices to effectuate at least one 
of the mood, focus, or intent (column 10 lines 30-55). 
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8. Regarding claim 5, in addition to that mentioned for claim 1 , Strubbe et al also 
show the execution of the actions includes controlling devices to request further user 
input to assist in making the determinations (column 1 1 lines 30-50). 

9. Regarding claim 6, in addition to that mentioned for claim 1 , Strubbe et al also 
show the execution of the actions includes initiating a process to at least one of 
complete, correct, and disambiguate what the system understands from previous input 
(column 10 lines 10-40). 

10. Regarding claim 7, in addition to that mentioned for claim 1 , Strubbe et al also 
show the processor abstracting the received input data into one or more events prior to 
making the determination (column 19 lines 10-30). 

1 1 . Regarding claim 8, in addition to that mentioned for claim 1 , Strubbe et al also 
show the processor performing recognition operations on the received input data 
(column 19 lines 10-23). 

12. Regarding claim 9, in addition to that mentioned for claim 1 , Strubbe et al show 
the dialog manager which determines an object or topic to which a user is referring 
(column 17 lines 1-14). Furthermore, Geddes uses a dialog manager to determine 
intent to determine an object or topic to which a user is referring (column 7 lines 35-65). 
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It would have been obvious to a person with ordinary skill in the art to determine the 
user's intent in the system of Strubbe et al, because it would provide a convenient way 
to determine an object or topic to which a user is referring. 

13. Claims 10-17 show the same features as claims 1-8 respectively and are 
rejected for the same reasons. 

14. Claim 18 shows the same features as claim 1 and is rejected for the same 
reasons. 

15. Regarding claim 19, in addition to that mentioned for claim 1, the multi-modal 
data includes audio and image based (Strubbe et al Figure 3). 

16. Regarding claim 20, the intent determination in Geddes includes resolving 
referential ambiguity associated with one or more users in the environment based on a 
portion of the received data (Geddes column 8 lines 17-30). The obviousness to 
combine into Strubbe et al follows as in paragraph 4 of this Office Action. 

17. Regarding claim 21, the intent determination in Geddes includes resolving 
referential ambiguity associated with one or more devices in the environment based on 
a portion of the received data (Geddes column 7 lines 20-27). The obviousness to 
combine into Strubbe et al follows as in paragraph 4 of this Office Action. 
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18. Regarding claim 22, the system has an image capturing device (Strubbe et al 
Figure 3, the video camera). 

19. Regarding claim 23, the image data is visible (Strubbe et al Figure 3, inherent in 
the video camera). 

20. Regarding claim 24, the image data may be video (Strubbe et al Figure 3, the 
video camera). 

21 . Regarding claim 25, the system has audio capturing input devices (Strubbe et al 
Figure 3, the element 245). 

22. Regarding claim 26, the audio capturing device has a microphone (Strubbe et al 
column 21 lines 1-10). 

23. Regarding claim 27, the system has graphical user interface input devices 
(Strubbe et al column 20 lines 65-67, Figure 3). 

24. Regarding claim 29, in addition to that mentioned for claim 19, Strubbe et al also 
show the execution of the actions includes controlling devices to effectuate at least one 
of the mood, focus, or intent (column 10 lines 30-55). 
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25. Regarding claim 30, in addition to that mentioned for claim 19, Strubbe et al also 
show the execution of the actions includes controlling devices to request further user 
input to assist in making the determinations (column 1 1 lines 30-50). 

26. Regarding claim 31 , in addition to that mentioned for claim 19, Strubbe et al also 
show the processor abstracting the received input data into one or more events prior to 
making the determination (column 19 lines 10-30). 

27. Regarding claim 32, in addition to that mentioned for claim 19, Strubbe et al also 
show the processor performing recognition operations on the received input data 
(column 19 lines 10-23). 

28. Regarding claim 33, Strubbe et al show speech recognition (column 8 lines 62- 
67). 

29. Regarding claim 34, Strubbe et al show speaker recognition (column 8 lines 55- 
61). 

30. Regarding claim 36, in addition to that mentioned for claim 19, Strubbe et al also 
show the execution of the actions includes initiating a process to at least one of 
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complete, correct, and disambiguate what the system understands from previous input 
(column 10 lines 10-40). 

31 . Regarding claim 37, in addition to that mentioned for claim 19, Strubbe et al 
show the dialog manager which determines an object or topic to which a user is 
referring (column 17 lines 1-14). Furthermore, Geddes uses a dialog manager to 
determine intent to determine an object or topic to which a user is referring (column 7 
lines 35-65). It would have been obvious to a person with ordinary skill in the art to 
determine the user's intent in the system of Strubbe et al, because it would provide a 
convenient way to determine an object or topic to which a user is referring. 

32. Claim 38 shows the same features as claim 19 and is rejected for the same 
reasons. „ 

33. Claims 28 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Strubbe et al (6731307) and Geddes (6751661) and Adler (6675356). 

34. Regarding claim 28, in addition to that mentioned for claim 19, Strubbe et al and 
Geddes mention a variety of input devices for graphical user interface systems for 
convenient inputting of data, but do not mention the stylus input device. Furthermore, 
Adler et al show a stylus input device for capturing handwritten data, used in graphical 
user interface systems, for convenient inputting of data (column 7 lines 25-30). It would 



Application/Control Number: 09/776,654 Page 9 

Art Unit: 21 74 

have been obvious to a person with ordinary skill in the art to have the stylus input 
device in the invention suggested by Strubbe et al and Geddes, because it would allow 
convenient inputting of data in a graphical user interface. 

35. Regarding claim 35, in addition to that mentioned for claim 32, Strubbe et al and 
Geddes mention a variety of input devices for graphical user interface systems for 
convenient inputting of data, but do not mention the stylus input device. Furthermore, 
Adler et al show a stylus input device for capturing handwritten data, used in graphical 
user interface systems, for convenient inputting of data (column 7 lines 25-30). It would 
have been obvious to a person with ordinary skill in the art to have the stylus input 
device in the invention suggested by Strubbe et al and Geddes, because it would allow 
convenient inputting of data in a graphical user interface. 

36. Applicants arguments with respect to the claims have been considered but are 
moot in view of the new ground(s) of rejection. However, note that in view of the new 
art, the finality has been withdrawn. 

37. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Steven P Sax whose telephone number is 571-272- 
4072. The examiner can normally be reached on M-F 8:30AM - 5:00PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kristine Kincaid can be reached on 571-272-4063. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 



*** 



STEVEN SAX 



